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i.o  introduction 

In  FLIR,  as  in  other  high  dynamic  range  images,  the  dynamic  range  of 
the  scene  frequently  'exceeds  that  of  the  display.  Scenes  containing  bright 
objects,  or  the  horizon,  require  the  operator  to  readjust  the  display 
parameters  for  each  portion  of  the  scene.  Under  rapidly  changing  image 
condition,  the  operator  with  all  his  other  duties  is  not  able  to  respond 
and  is  thus  forced  to  leave  the  settings  in  a nominal  position,  thus  view- 
ing nearly  everything  under  non-optimum  conditions. 

Proper  image  enhancement  techniques  can  intelligently  decrease  the 
total  dynamic  range  of  the  image  data  by  reducing  those  portions  of  the 
signal  which  do  not  contribute  toward  target  recognition  and  identification 
while  preserving  and  enhancing  target  related  features. 

The  aim  of  this  effort  has  been  to  evaluate  Rockwell's  previously 
developed  locally  developed  contrast  normalization  and  other  image  enhancement 
techniques  to  FLIR  image  processing.  The  local  contrast  enhancement 
technique  has  demonstrated  the  ability  to  match  high  dynamic  range  image  data 
to  a limited  dynamic  range  display.  Additional  image  enhancement  techniques 
investigated  include  contouring  , one-  and  two-dimensional  frequency  filtering, 
and  interframe  averaging. 

t 

The  degree  of  image  enhancement  obtained  was  strongly  affected  by  the 
signal-to-noise  ratio  of  the  available  data.  Target/background  to  peak-to- 
peak  noise  ratios  of  less  than  one  were  frequently  experienced.  Linear  and 
nonlinear  noise  reduction  computer  programs  were  generated  for  the  removal  of 
salt  and  pepper  noise,  correlated  noise,  consisting  of  various  types  of 
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ringing  as  well  as  fiduciary  marks.  The  programs  proved  effective  in 
significantly  reducing  the  effects  of  these  noises.  Subsequent  appli- 
cation of  the  image  enhancement  programs  demonstrated  the  ability  to 
improve  background  contrast  ratios  without  saturating  any  target  charac- 
teristics and  thus  make  the  background  more  clearly  visible. 

Dynamic  contrast  enhancement  appears  to  be  an  effective  means  for 
freeing  the  operator  from  continual  display  system  adjustments.  Addi- 
tional optimization  and  testing  on  image  data  obtained  more  closely 
from  the  focal  plane  and  containing  very  hot  targets  and  contrasting 
backgrounds,  such  as  horizons,  is  recommended  to  more  fully  evaluate 


the  technique. 
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SUMMARY  OF  RESULTS 


To  paraphrase  an  old  saying;  "A  picture  is  worth  a thousand  words 
and  a million  numbers."  Indeed,  the  collection  of  pictures  contained  in 
this  section  is  the  best  way  to  summarize  the  results  of  this  FLIR  digital 
image  enhancement  study.  Specific  algorithms  applied  are  referenced  by 
entries  in  Table  2-1  and  parameter  values  are  found  in  the  "operation  notes" 
listing. 


2.1  Display  Characteristics 

The  CRT/ film  characteristics  and  the  display  software  merit 
discussion,  for  if  uncontrolled  their  effects  on  the  picture  can  overwhelm 
the  most  skillfully  developed  enhancement  algorithm.  A test  image  and  its 
histogram  found  in  Figure  2-0  illustrate  ten  uniformly  spaced  grey  levels 
which  span  the  output  system's  dynamic  range.  The  display  software  normalized 
image  point  values  to  produce  pictures  which  utilized  this  range.  The 
operator  provided  the  values  MAXB  and  MINB  of  equation  2.1  based  upon  the 
histogram  of  the  image  to  be  photographed. 


[ L MAXB  -minbJ  1 


Eq.  (2.1) 


The  brightness  and  bias  parameters  W and  B were  set  such  that  B volts 
provided  the  control  signal  to  generate  the  darkest  nonsaturated  intensity 
and  W volts  the  brightest  nonsaturated  intensity. 
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TABLE  2-1. 

PHOTOGRAPH 

DESCRIPTIONS  AND 

CROSS  REFERENCE  GUIDE 

Fi gure 

Description 

Figure 

Processed 

Operation 

Note 

Technique  Illustrated 

Paragraph 

Reference 

2-1 

APC/E; Tank/ 3/4 

Orig 

. 

. 

_ 

2-2 

II 

2-1 

1. 

Nonlinear  filtering 

3.1.1 

2-3 

If 

2-2 

2. 

II 

3.1.1 

2-4 

II 

2-3 

3. 

Dynamic  Contrast  Norm. 

3.2.1 

2-5 

II 

2-3 

4. 

II 

3.2.1 

2-6 

If 

2-1 

5. 

Nonlinear  filtering 

3.1.1 

2-7 

II 

2-6 

6. 

Nonlinear  8 lowpass  fil- 

3.1.3.1 

2-8 

If 

2-6 

7. 

tering 

II 

3.1 .3.1 

2-9 

II 

2-8 

8. 

Dynamic  Contrast  Norm. 

3.2.1 

2-10 

II 

2-8 

9. 

II 

3.2.1 

2-11 

II 

2-10 

- 

Logrithmic  Pt.  Function 

- 

2-12 

Histogram 

2-14 

- 

_ 

- 

2-13 

II 

2-15 

- 

- 

- 

2-14 

APC/E 

Orig. 

- 

- 

- 

2-15 

II 

2-14 

10. 

Coarse  Quantization 

- 

2-16 

II 

2-14 

11. 

Nonlinear  filtering 

3.1.1 

2-17 

II 

2-16 

10. 

Coarse  Quantization 

- 

"’-18 

II 

2-16 

12. 

Dynamic  Contrast  Norm. 

3.2.1 

19 

II 

2-16 

13. 

II 

3.2.1 

2-20 

If 

2-16 

14. 

II 

3.2.1 

2-21 

Histogram 

2-18 

- 

- 

- 

2-22 

II 

2-19 

- 

- 

- 

2-23 

II 

2-20 

- 

- 

2-24 

Tank/ 3/ 4 

Orig. 

- 

_ 

- 

2-25 

Histogram 

2-24 

- 

- 

- 

2-26 

Power  Spectrum 

2-24 

- 

- 

- 

2-27 

Tank/ 3/4 

2-24 

15. 

Linear  Bandstop  Filter 

3. 1.3. 2 

2-28 

Histogram 

2-27 

- 

- 

- 

2-29 

2-30 

Tank/S; 2*2  Ton/S 

If 

Orig. 

2-29 

16 

Inter-frame  Averaging 

3.1.4 

2-31 

II 

2-29 

17 

II 

3.1.4 

2-32 

2l->;  Tank;  APC 

Orig. 

- 

- 

- 

2-33 

ii 

Orig. 

- 

- 

- 

2-34 

n 

Orig. 

- 

- 

- 

2-35 

32x64  Sub-element 

2-32 

- 

- 

- 

2-36 

ii 

2-33 

- 

- 

- 

2-37 

n 

2-34 

- 

- 

- 
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Table  2-1.  (Continued) 


igure 

Description 

Figure 

Processed 

Operation 

Note 

Technique  Illustrated 

Paragraph 
Re ference 

2-38 

2Js;  Tank;  APC 

2-32  to  2-34 

18 

Inter-frame  Averaging 

3.1.4 

2-39 

II 

II 

19 

II 

3.1.4 

2-40 

If 

2-32 

20 

Dynamic  Contrast  Norm. 

3.2.1 

2-41 

11 

2-39 

20 

II 

3.2.1 

2-42 

2k;  Tank;  APC 

2-32,2-33 

21 

Difference  W/0  Offset 

_ 

2-43 

M 

2-32,2-34 

21 

II 

- 

2-44 

ft 

2-32,2-33 

22 

Difference  with  Offset 

- 

2-45 

ft 

2-32,2-34 

22 

11 

- 

2-46 

Aircraft  Carrier 

Orig. 

- 

- 

- 

2-47 

11 

2-46 

20 

Dynamic  Contrast  Norm. 

3.2.1 

2-48 

11 

2-47 

- 

Logarithmic  Function 

- 

2-49 

If 

2-46 

- 

II 

- 

2-50 

»» 

2-49 

20 

Dynamic  Contrast  Norm. 

3.2.1 

2-51 

2-52 

Plane  and  Ground 

II 

Orig. 

2-51 

20 

Dynamic  Contrast  Norm. 

3.2.1 

2-53 

ft 

2-52 

- 

Logarithmic  Function 

- 

2-54 

2-55 

Plane  and  Sky 

If 

Orig 

2-54 

20 

Dynamic  Contrast  Norm. 

3.2.1 

2-56 

II 

2-55 

- 

Logarithmic  Function 

'>-57 

Line  Scan  IR 

Orig. 

- 

- 

- 

-58 

If 

2-57 

- 

Logarithmic  Function 

- 

2-59 

II 

2-57 

23 

Dynamic  Contrast  Norm. 

3.2.1 

2-60 

If 

2-57 

24 

11 

3.2.1 

2-61 

If 

2-57 

25 

II 

3.2.1 

2-62 

II 

2-57 

26 

11 

3.2.1 

2-63 

APC/E;  Tank/3 A 

Orig. 

- 

- 

- 

2-64 

If 

2-63 

- 

Band  Emphasis 

3.2.2 

2-65 

If 

2-64 

- 

Contouring 

3.2.2 

2-66 

25-;  Tank;  APC 

Orig. 

- 

- 

- 

2-67 

II 

2-66 

- 

Band  Emphasis 

3-2.2 

2-68 

II 

2-67 

- 

Contouring 

3.2.2 

2-3 
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OPERATION  NOTES 


1. 


2. 

3. 

4. 

5. 

6. 


7. 


8. 

9. 

10. 

11. 

12. 

13. 

14. 

15. 


16. 


DESPIKEl  Cl,  1,  5,  5,  100,  0) 

DEMESA  (1000,  100,  5,  1,  1,  1,  1500,  1500) 
CHOP  (10) 

CHOP  (10) 

DEMESA  (500,  0,  75,  1,  1,  1,  4000,  4000) 
Transpose  image  and  repeat  1. 

IMSHADE  (1,  50,  50,  50,  50,  10000) 

IMSHADE  (1,  50,  50,  80,  80,  10000) 

CHOP  (10) 


Operation  X 

LOWPASS  (50,  10) 
Transpose  Image 
Operation  X 
LOWPASS  (50,  5) 


CHOP  (10) 

DEMESA  (1000,  1,  5,  1,  1,  1,  1500,  1500) 
CHOP  (10) 

DEMESA  (500,0,  75,  1,  1,  1,  4000,  4000) 


Operation  X 
LOWPASS  (50,  20) 

Transpose  Image 
Operation  X 
LOWPASS  (50,  10) 

IMSHADE  (1,  0,  0,  90,  90,  10000) 

IMSHADE  (1,  20,  20,  75,  75,  10000) 

A(i,j)  = [A(i,j)/1000]  • 1000;  (IntegerArithmetic  and  MAX  (A(i , j ) ) - 100UC) 
CHOP  (10) 

DESPIKEl  (2,  2,  10,  10,  10,  100,  0) 

IMSHADE  (1,  0,  0,  50,  50,  10000)  followed  by  same  quantization  as  operation  10 


IMSHADE  (1,  0,  0,  75,  75,  10000)  followed  by  same  quantization  as  operation  10 


IMSHADE  (1,  0,  0,  100,  100,  100,  10000)  followed  by  same  quantization  as 
operation  10. 


Linear  Bandstop  Filter  Program 

Frequencies  passed:  0 .<  f I 1/32  (X  ^ 32) 

1/12  t'  f i 1/6  (12  5s  X > 6) 

Image  values  replaced  by  mean  of  clustered  data  point  values  from  three  TV 
frames  using  DESPIKEl  algorithm.  r\^  = 1,  n^  = 10,  n^  = 10 
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17.  Image  values  replaced  by  mean  of  clustered  data  point  values  from  five 
TV  frames  using  DESPIKEl  algorithm.  n7  = 2,  n^  = 10,  n^  = 10. 

18.  Same  as  operation  16  except  n^  = 1 and  n^  = 1 . 

19.  Same  as  operation  16  except  n9  = 2,  n^  = 1,  and  n^  = 50. 

20.  IMSHADE  (1,  0,  0,  99,  99,  10000) 

21.  A(i,j)  = - A2(i,j)  + 5000 

22.  A(i , j ) = A1(i;j)  - A2(i  + kQ,j  + lQ)  + 5000 

23.  IMSHADE  (1,  5,  5,  90,  50,  10000) 

24.  IMSHADE  (1,  0,  0,  90,  90,  10000) 

25.  IMSHADE  (1,  0,  0,  95,  95,  10000) 

26  IMSHADE  (1,  1,  1,  99,  99,  10000) 
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IMAGE  RESTORATION  AND  ENHANCEMENT  TECHNIQUES 


Image  processing  algorithms  fall  into  two  categories  - restoration 
and  enhancement.  Restoration  algorithms  seek  to  undo  the  effects  of  all 
noise  sources  in  the  imaging,  digitizing,  and  transmitting  processes. 
Enhancement  algorithms  are  concerned  with  the  observer  and  seek  to  display 
the  scene  information  optimally  for  the  human's  visual  system.  They 
deliberately  distort  the  restored  image  so  as  to  emphasize  selectively 
features  most  likely  to  help  the  observer  perform  his  task. 


reduce  the  ringing  effects. 

One  dimensional  algorithms  operated  sequentially  on  scan  lines. 
Transposition  of  the  image  arrays,  however,  permitted  application  of  these 
techniques  along  both  vertical  and  horizontal  directions.  Two-dimensional 
algorithms  were  local  operations  and  required  only  the  eight  nearest  neighbors' 
image  point  values.  In  all  local  one  and  two-dimensional  nonlinear  algorithms 


The  algorithms'  effects  are  illustrated  by  cross-section  plots  of 
the  £mage  intensity  functions  and  power  spectral  density  plots.  Figure  3-1 
contains  the  image  data  which  served  as  the  common  initial  values  for  the 
cross-sections  and  power  spectrum  plots  that  follow. 

3.1.1  Nonlinear,  One-Dimensional,  Local  Techniques 

The  CHOP,  DESPIKE  1,  and  DEMESA  algorithms  were  developed  to 
suppress  spike  noise  and  data  marks.  Pictorial  results  are  illustrated  in 
Figures  2-2,  2-3,  2-6,  and  2-l6. 

3. 1.1.1  CHOP  (Parameter  = N) 

Concept : Image  point  values  which  are  extrema  with  respect  to 

their  nearest  neighbors  are  recomputed.  The  increments  are  a specified 
proportion  (N/10)  of  the  difference  of  the  original  image  point  value  and 
of  one  of  its  nearest  neighbors. 

Algorithm : 

If  A(i-l)  S A(i)  < A(i+1) 

If  A(i-l)  > A(i)  > A( i+1 ) 

If  A(i-l)  < A(i)  > A( i+1 ) 

If  A(i-l)  > A(i)  < A( i+1 ) 


A(i)  = A(i) 

A(i)  = A(i) 

A(i)  = A( i ) - [A( i )-Max(A( i+1 ) ,A(i-l ) ] 

A( i ) = A(i)  - [A( i )-Min(A( i+1 ) ,A( i-1 ) ] 
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Application:  CHOP'S  usefulness  is  limited  to  the  suppression  of 

single  pixel  spike  noise.  If  its  input  parameter  (N)  is  set  greater  than 
10,  negative  image  point  values  can  result.  Setting  it  equal  to  10  did  the 
best  job. 

Although  only  a one-dimensional  version  of  the  concept  was  imple- 
mented, it  could  easily  be  implemented  as  a two-dimensional  algorithm  using 
the  surrounding  eight  rather  than  the  horizontally-adjacent  two  nearest 
neighbors. 

Example  (N=10) 

A(i) 


n » n - , » n * n^  * ^6^ 


3, 1.1.2  DESPIKE1  (Parameters  = 

C ncept:  Image  p int  values  are  replaced  with  a weighted  average 

f their  initial  values  and  the  mean  values  of  a clustered  set  of  neighbor 
points . 


Parameter  definitions 


n,  - Running  mean  window  = A.  , A.. 

1 l-n  i+n, 

n,  - Limit  to  number  of  points  deleted 

in  computation  of  clustered  local  mean. 

n^  - Threshold  for  maximum  positive  percent 
deviation  from  mean. 

n,  - Threshold  for  maximum  negative  percent 
deviation  from  mean. 


n 


5 


Weighting  coefficient  used  in  update  calculation 
if  the  original  point  is  excluded  from  the  mean  set. 


n6 


Weighting  coefficient  used  in  update  calculation 
if  original  value  included  in  clustered  mean  set. 


Algorithm 


Step  1 - Compute  mean  in  running  window  determined  by  n . 

Step  2 - Delete  one  point  (if  any)  which  lies  outside  limits 
set  by  n^  and  n^.  (Measured  in  units  of  n^  and  n^ 

values).  If  no  points  fall  outside  the  range  speci- 
fied , skip  ahead  to  Step  5. 

Step  3 - Recompute  mean  value  l(i,  n^ ,n0 ,n^,n^ ) . 

Step  k - If  number  of  points  deleted  does  not  exceed  n^, 
repeat  steps  1,  2,  and  3. 

Step  5 - Recompute  l(i)  as  follows: 

I’(i)  = (l-o)l(i)  + of.l(i,n1,n2,n3,nlt) 

If  l(i)  deleted  in  computation  of  I,  then  a = n^ . 

If  l(i)  not  deleted  in  computation  of  I,  then  a = 
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Application 


The  DESPIKE  algorithm  is  the  most  versatile  of  the  one-dimensional 
nonlinear  algorithms  developed.  Particularly  useful  are  the  separate 
threshold  percentages  (n^  and  n^)  for  positive  and  negative  deviations  from 
the  cluster  mean  and  the  separate  weighting  coefficients  (n^  and  n^)  which 
allow  completely  independent  updating  of  an  image  point  depending  upon  its 
deviation  from  the  local  clustered  mean.  The  two  thresholds  provide  in- 
dependent sensitivity  to  positive  and  negative  spike  noise  and  the  two 
coefficients  provide  a more  selective  update  of  the  image  point  values. 


3. 1.1. 3 DEMESA  (Parameters  = n^,  n 2,  n^,  n^ , n,.,  n^,  n^,  ng) 

Concept : DEMESA  operates  on  the  one-dimensional  derivative 

obtained  by  taking  first  differences:  I'(i,j)  = l( i+1, j )-I ( i , j ) . It 

searches  for  paired  positive  and  negative  peaks  which  exceed  a specified 
threshold  and  which  lie  within  a specified  neighborhood.  The  derivative 
for  all  points  in  these  paired  peak  regions  is  replaced  by  the  mean  of  the 
original  derivative  function  values.  The  objective  of  the  algorithm  is  to 
eliminate  spike  noise  and  features  with  sharp  edges  (e.g.,  data  markings). 


Algorithm 

Step  1 - Replace  image  intensity  function  by  its  derivative. 

Step  2 - Perform  a forward  search  for  a point  whose  absolute 
value  exceeds  n^.  Go  to  Step  7 if  end  of  scan  line 
is  encountered. 

Step  3 - Search  forward  and  backward  from  the  point  found  in 
Step  2 until  the  function  either  changes  polarity  or 
until  its  absolute  value  is  less  than  n^.  Compute  the 
sum  of  the  values  of  the  set  of  points  within  the 
region  defined  by  this  search.  If  the  sum's  absolute 
value  is  less  than  n^,  return  to  Step  2. 
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Step  H - Perforin  another  forward  search  for  a distance  not  to 

exceed  n^  until  a functional  value  is  found  of  opposite 
polarity  and  whose  absolute  value  exceeds  n^.  If  no 
such  point  is  found,  return  to  Step  2. 

Step  5 - Repeat  the  operation  described  in  Step  3 in  the  neighbor- 
hood of  the  peak  of  opposite  polarity.  If  the  sum's 
absolute  value  is  less  than  ng,  return  to  Step  h. 

Step  6 - Compute  the  average  of  the  functional  values  of  all  points 
contained  in  the  two  regions  defined  by  the  searches  of 
Steps  3 and  5-  Replace  each  of  the  values  by  this  average 
and  return  to  Step  2. 

Step  7 " Integrate  the  updated  derivative  function  and  proceed  to 
the  next  scan  line. 

Options:  (l)  To  skip  Steps  1 and  7,  set  n^  = 0 

(2)  To  iterate  N times,  set  n^  = N 

(3)  To  search  only  for  positive  peaks  in  Step  2, 
set  ng  = 1. 

Application 

DEMESA's  use  was  limited  to  removal  of  data  markings  which  inter- 
ferred  with  the  dynamic  contrast  normalization  enhancement  algorithm  IMSHALE. 
Cross-sectional  plots  of  the  image  intensity  function  through  targets  and 
data  marks  were  studied  to  determine  the  values  of  DEMESA's  parameters  that 
would  selectively  remove  the  data  marks  without  affecting  the  target. 


n = 500 
n2  = I 
n3  = 10 

ni,  = 1 

n5  = 1 
n6  = 1 

rij  = 500 

ng  = 500 


(threshold) 

(secondary  threshold) 

(search  distance) 
(differentiate) 

(number  of  passes) 

(trigger  on  positive  spikes) 
(area  threshold,  first  peak) 
(area  threshold,  second  peak) 


Parameter  values  for  DEMFSA  Examples  1 and 


Figure  3-5  DEMESA  Example  ? (Same  Parameter  Values  as 

Example  l) 


3.1.2  Nonlinear,  Two-Dimensional,  Local  Techniques 
3.1.2. 1 DESPIKE2  (Parameters  = n , n ) 

Concept : The  image  point  value  is  replaced  "by  a weighted  average 

of  its  initial  value  and  the  mean  of  8-N  of  its  surrounding  eight  neighbors. 
The  N values  excluded  from  the  neighborhood  mean  are  those  which  are  re- 
jected in  the  following  clustering  algorithm. 

Algorithm 

Step  1 - Set  m = 0 

Step  2 - Form  average  M of  8-m  nearest  neighbors. 

Step  3 - Compare  absolute  value  of  the  8-m  differences 
between  the  average  of  Step  2 and  image  point 
values.  Eliminate  from  the  nearest  neighbor  set 
that  point  with  the  largest  deviation. 

Step  1 - Set  m = m+1 

Step  5 - If  n < m,  return  to  Step  2 

Step  6 - Compute  a new  value  for  the  image  point  according 
to  the  following  expression 

n?  I - 1 

I'(i,j)  = l(i, j ) + YOO  ‘ I M " I(i’j)| 

Application 

DESPIKE  2 was  used  to  suppress  single  pixel  and  single  scan  line 
noise  from  the  images.  The  specific  algorithm  described  is  representative 
of  many  nearest  neighbor  nonlinear  filter  functions  which  have  straight- 
forward hardware  implementations.  They  defy  conventional  analysis  and  are 
based  on  intuition  and  heuristic  reasoning. 
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3.1.3 


Linear,  One-Dimensional  Techniques 


The  LOWPASS  and  FOURFILT  filtering  programs  removed  frequency- 
components  in  the  image  function  in  which  noise  power  was  significant. 
Pictorial  results  are  found  in  Figures  2-7,  2-8,  and  2-27. 

3.1. 3.1  LOWPASS  (Parameters  = L , X.  ) 

Concept : A conventional  one-dimensional  lowpass  filtering 

approach  was  tried  on  the  FLIR  images  to  suppress  high  frequencies . 

The  high  frequency  content  of  the  digitized  images  must  be  due  to  additive 

noise  sources  because  the  video  disk's  6 MHz  bandwidth  does  not  support  the 

number  of  samples  taken  along  each  scan  line.  This  oversampling  effect  is 

seen  in  plots  of  one  scan  line's  102k  sampled  image  points. 

Algorithm 

Step  1 - Compute  2L  + 1 terms  of  the  inverse  Fourier  transform 
of  an  ideal  lowpass  filter  with  cutoff  frequency  1/\Q- 

h(  i ) = ^ sin  • — — for  - L < i < L 

L <50 

Step  2 - Extend  the  image  function  beyond  its  borders  by 
generating  its  mirror  image. 

l(-i)  = l(i+l)  for  0<i<L 
l(N+l+i)=l(N-i)  for  0<i<L 

Step  3 - Obtain  the  filtered  image  function  from  the  convolution 

sum  below.  T 
Li 

I ' (i)  = s I(i-j)  • h(j) 

j=-L 
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Applications 

The  bandwidth  of  the  noise  proved  to  he  so  broad  that  significant 
amounts  of  ringing  remained  at  cutoff  frequencies  for  which  both  target  and 
background  details  were  blurred. 

Examples : 

FILTERED:  L = 50;  X = : 

8000- 

1+000  - 
1000  - 

\ I | t 1 

0 100  200  x 300  l+oo  500 


Figure  3-6  Lowpass  Example  1 
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3.1. 3.2  FOURFILT 


The  failure  01  LOWPASS  to  remove  the  ringing  noise  without  un- 
acceptable blurring  of  target  and  background  objects  led  to  consideration 
of  a conventional  linear  filter  with  stop  bands  in  those  regions  of  the 
frequency  domain  that  were  dominated  by  noise.  Determination  of  those 
regions  was  based  upon  inspection  of  the  power  spectral  density  function 
of  the  image.  Sequential  scan  lines  were  concatenated  to  compute  the 
average  power  density  in  a horizontal  slice  of  the  image.  A stop  band 
was  placed  where  a peak  in  the  function  suggested  the  presence  of  noise 
power.  High  frequency  components  were  also  removed.  Cross-section 
plots  through  the  target  regions  of  Figures  2- 2k  and  2-27  are  found  in 
Figure  3-10  and  3-11.  A dramatic  improvement  in  the  signal-to-noise 
ratio  was  achieved  by  this  filtering  method. 

3.1.k  Inter frame  Averaging  Experiments 

All  of  the  restoration  techniques  which  have  been  described  thus- 
far  perform  spatial  filtering  on  single  frames.  Another  approach  that 
takes  advantage  of  the  large  difference  in  the  temporal  correlation 
properties  of  the  signal's  scene  and  noise  components  is  interframe 
averaging.  Since  the  scene  intensity  varies  in  time  more  slowly  than 
much  of  the  noise,  performing  low  pass  temporal  filtering  can  improve  the 
signal-to-noise  ratio.  A consecutive  sequence  of  images  taken  by  a FLIR 
system  was  available  to  test  this  approach  to  restoration. 

Method : The  first  step  was  to  determine  the  horizontal  and 

vertical  shifts  of  the  target  between  successive  TV  frames.  A 32x61+  sub- 
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Figure  3-11.  Bands top  filtered,  cross-section  through  target 

(Refer  to  Figure  2-27 ) 


element  containing  the  target  in  the  first  frame  of  the  TV  sequence  served 


! 


1 


as  a matching  template.  A simple  metric  was  computed  between  the  template 
and  32x64  subelements  in  subsequent  frames.  A matrix  of  these  scores  was 
generated  by  displacing  the  template  within  a region  ±14  pixels  horizontally 
and  vertically  from  nominal.  The  metric  is  defined  by  the  following 
equations: 


32  64 

Mn(k,4)  = Z 2 mn(i,j,k,f) 
i=l  j=l 


mn(i,j,k,f)  = 1 if  | T(i,j )-An(i+io+  k,j+jo+4)|<  t 

mn(i,J,k,l)  = 0 if  |T(i  j)-An(i+io+k,j+Jo+f )|  > t 

where  - l4  < k < l4 
- 14<  l <14 


Jo  Jo  K 

1 

J “ 

— r~? 

1 — f\ 

J i 

a.0+  k 

1 

V 

Frame  1 Frame  n 


o 


The  desired  horizontal  and  vertical  shifts  were  obtained  from  the  indicies 
of  the  maximum  value  within  the  M array.  No  rotations,  scale  changes,  or 
perspective  changes  were  taken  into  account  in  these  experiments. 

The  second  step  was  to  create  a composite  image  by  superimposing 
several  images  and  computing  new  image  point  values.  Were  a Gaussian  white 
noise  model  assumed,  simple  averaging  would  be  the  optimum  calculation  to 
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perform.  The  spike  noise  observed  failed  to  fit  this  model,  however,  and 
a DESPIKE1  based  algorithm  (see  para.  3. 1.1. 2)  was  applied  to  compute  an 
average  of  a clustered  set  of  image  points. 

Results : 

Inter frame  averaging  experiments  were  performed  on  two  sets  of 
FLIR  images.  The  first  set  consisted  of  five  consecutive  frames  of  a TV 
sequence.  The  first  frame  and  two  composite  pictures  of  the  scene  are 
found  in  Figures  2-29  to  2-31.  The  first  composite  (Fig.  2-30)  was  obtained 
from  a point-by-point  average  of  either  two  or  three  image  points  from  three 
consecutive  frames.  All  three  points  were  averaged  if  no  one  point  deviated 
by  more  than  10  percent  from  the  mean.  Otherwise,  the  two  values  nearest  the 
mean  were  averaged  to  compute  the  composite  image's  value.  The  second  com- 
posite (Fig.  2-31)  was  obtained  from  a similar  calculation  on  five  consecutive 
frames.  The  only  difference  was  that  as  many  as  two  values  could  be  rejected 
during  the  averaging  procedure  if  their  values  deviated  by  more  than  10  per- 
cent from  the  mean.  Images  generated  by  taking  pointwise  differences  (and 
adding  a constant  to  avoid  negative  functional  values)  are  displayed  in  Figures 
2-^2  to  2-l5-  The  top  two  were  formed  without  the  spatial  offset  required  for 
superimposing  the  32x6k  subelements  already  described.  The  bottom  two  were 
formed  from  differences  of  appropriately  offset  image  functions. 

The  second  set  consisted  of  three  nonconsecutive  frames.  Perspective 
changes  caused  objects  beyond  the  vicinity  of  the  matching  template  to  blur 
badly.  Figures  2-32  to  2-37  contai.i  the  three  frames  and  enlarged  views  of 
the  equivalent  subelements  selected  by  the  cross-correlation  routine.  Two 
composite  versions  are  shown  in  Figures  2-38  and  2-39- 
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The  first  composite  was  obtained  from  a pointwise  average  of  two 
image  points  from  the  three  frames.  The  two  values  averaged  were  those 
most  nearly  equal  to  the  mean  of  all  three  values.  The  second  composite 
was  obtained  by  selecting  the  minimum  value  of  the  three  image  point  values 
from  the  three  frames.  Observe  that  the  data  marks  are  almost  totally 
absent  in  the  second  composite  (Fig.  2-39)  because  the  frame-to-frame  off- 
sets were  so  large  and  the  minimum  (i.e.,  darkest)  image  value  of  the  three 
frames  was  selected  for  the  composite  image. 


Conclusions : 

Interframe  averaging  improved  the  FLIR  image  signai-to-noise  ratio 
(see  plots  below).  However,  the  available  imagery  (Figure  2-2l)  which 
represented  successive  TV  rames  lacked  sufficient  scene  detail  to  evaluate 
-he  perceptual  effects.  The  "busier"  scene  (Figs.  2-32  to  2-3M  imagery  did 
not  represent  successive  TV  frames  and  perspective  changes  between  images 
badly  blurred  objects  away  from  the  registered  subimager. 

Single  Frame  Average  of  5 Frames 

Slice  Through  Tank  Slice  Through  Tank 

10,000  10,000 

8000.  8000  - 


6000 

I(X,386) 

k000- 


6ooo 

I(X,386) 

1000 


2000  ' 


2000 


0 1 I i • 

I486  511  536  561 

3-17 


0 , 1 1 1 

186  511  536  561 


3.2 


Image  Enhancement  Techniques 


3.2.1  Dynamic  Contrast  Normalization  - IMSHADE 

A dynamic  contrast  normalization  algorithm  (IMSHADE),  which  has 

demonstrated  its  enhancement  capability  on  natural  scenes,  was  the  primary 

enhancement  method  evaluated  with  FLIR  images.  IMSHADE  is  a compromise 

between  contrast  normalization  schemes  which  are  based  solely  on  the  local  or 

global  range  of  the  image  intensity  function.  It  generates  coarse  ceiling 
and  floor  arrays  from  local  extrema  and  then  partially  "relaxes"  these  limits 

toward  the  scene's  global  extremes.  Consequently,  local  object-background 

contrast  is  enhanced  without  losing  global  background  variation.  Its  most 

impressive  results  are  obtained  when  global  background  differences  are  on 

the  same  order  as  local  object-background  differences.  Enhancement  examples 

of  non-FLIR  images  are  found  in  Figures  2-h6  through  2-62. 

Many  examples  of  IMSHADE-processed  FLIR  images  are  found  in  Section  2. 
The  poor  signal-to-noise  ratio  and  the  relatively  small  global  backgound  in- 
tensity variations  in  the  original  FLIR  images  complicate  the  evaluation  of 
these  results.  Nevertheless,  the  photographs  do  demonstrate  IMSHADES's 

capacity  to  increase  local  contrast  without  complete  loss  of  global  background 
variations . 

IMSHADE  Algorithm:  (Parameters  = N,  lOOcs^,,  lOOo^,  100(3„,  100(3^,  W) 

The  algorithm's  steps  are  introduced  by  way  of  a simplified,  one- 
dimensional example  in  Figures  3-12  through  3-17. 


3-18 


I(x) 


l(x)  is  the  intensity  (or  brightness)  of  a one-dimensional  image 
sampled  at  256  discrete  points.  (The  equations  to  be  described  are 
trivially  extended  to  the  two-dimensional  problem  encountered  in  practice.) 
If,  as  is  often  the  case,  the  size  of  interesting  objects  is  much  less  than 
the  full  field  of  view  (256  pixels  in  the  example),  then  what  appears  to  be 
an  edge  in  Figure  3-12  is  probably  a change  in  the  background  intensity 
(e.g. , the  horizon  of  an  IR  image).  The  smaller  amplitude  peaks  and  valleys 
are  probably  features  of  greater  interest  and  deserve  a greater  fraction  of 
the  limited  display  dynamic  range  than  they  possess  in  the  raw  image.  The 
algorithm  steps  to  be  described  achieve  that  goal  in  a controlled  way. 
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I(x) 


X 

Figure  3-13  - Ceiling  and  Floor  Arrays  C(i)  and  F(i) 

Defined 

Eight  "sub- images,"  each  comprised  of  32  pixels,  are  defined. 

The  subimage  dimensions  are  pre-selected  to  be  about  the  size  of  inter- 
esting features.  The  intensities  of  the  brightest  and  least  bright  pixel 
from  each  subimage  are  extracted  and  stored  in  the  "ceiling"  and  "floor' 
arrays,  C(i)  and  F(i),  respectively.  The  index  "i"  defines  the  subimage. 
Two  piecewise  constant  functions  C(x)  and  F(x)  are  shown  in  Figure  3-13  to 
represent  these  arrays. 


I(x) 


c(i)  = i [c(m)  + C(i+1)] 


,01  *C(i)+c(i) 

C'(i)  = MAX  [ + ^ , C(i)  } 

/a  •p(i)+p(i) 

Similarly:  F'(i)  = MIN  | ~+°< » F(  i ) > 

F ' 


E xayv  pit  : 
oC  p ~ ) 


Figure  3-1 H - Nonlinear  Filtering  of  C(i)  and  F(i) 


Each  extrema  value  in  the  ceiling  and  floor  arrays  is  averaged 
with  the  mean  value  of  its  nearest  neighbors.  Coefficients  an  and  O' ^ 

control  the  weighting  of  the  two  terms  in  each  sum.  The  nonlinear  aspect 
of  the  filtering  is  that  an  original  value  is  retained  if  the  filtered  value 
does  not  drive  the  local  extrema  towards  the  global  extrema.  At  the  image 
boundaries,  array  elements  C(0),  C(T)S  F(0)  and  F(T)  are  copied  into  the 
exterior  of  the  image  (C(-l),  C(8),  F(-l),  and  F(8)  ) to  perform  the  average. 
Finally,  the  filtering  can  be  performed  a variable  number  of  times  N,  the 
first  argument  in  the  IMSHADE  parameter  list. 
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C" ( i ) = B,  • C'(i)  + (l-BJ  • MAX  { C'(0),  C'(l),  C* (7) } 

F" ( i ) = Bp  • F ' ( i ) + (1-Bp)  • MIN  { F ' ( 0 ) , F'(l),  F'(T)} 


i(x) 


Figure  3-15  - Contrast  Expansion  on  C'(i)  and  F'(i) 


The  procedure  illustrated  in  Figure  3-15  is  the  key  step  in  the 
IMSHADE  algorithm.  The  tight  envelope  around  the  image  function  l(X)  formed 
by  the  ceiling  and  floor  functions  is  relaxed.  Each  value  in  the  ceiling 
array  C(i)  is  increased  a fixed  fraction  3^  of  the  difference  between  its  old 
value  and  the  maximum  value  in  the  array.  A similar  computation  is  performed 
on  each  element  of  the  floor  array  F(i).  This  procedure  allow  the  global 
extrema  in  l(X)  to  affect  the  normalization  bounds  of  every  image  point. 

The  global  extrema  are  often  determined  by  the  range  of  the  variation  in  the 
background's  intensity,  as  is  the  case  in  our  current  example,  and  this  step 
assures  that  background  perceptual  effects  will  be  partially  retained. 
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C(x ) = C(i)  + [c(i+l)-C(i) ]• 
F(x)  = F(i)  + [F(i+1)-F(i)]« 


where  i = MOD(X) 
m = R(X) 
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Figure  3-16  - Interpolation  of  C(i)  and  F(i) 

The  final  pre-normalization  step  is  to  create  piecewise  continuous 
ceiling  and  floor  functions.  This  step  prevents  the  boundaries  of  the  sub- 
images from  showing  in  the  enhanced  image. 


V7 

\ /V 


t'Iy)  = Itx)-F(x) 
IvXj  C(x)-F(x) 


0 32  6k  96  128  160  192  22k  256 

Figure  3-17  - Pointwise  Normalization  of  l(X) 

Each  image  point  value  is  recomputed  in  this  step.  The  upper  and 
lower  bounds  in  the  normalization  equation  vary  from  point  to  point.  This 
normalized  image  function,  depicted  for  our  example  in  Figure  3-17 , has  in- 
increased  local  object-background  contrast.  The  contrast  expansion  step,  however, 
has  assured  that  the  brightest  and  darkest  regions  in  the  original  image  continue 


to  be  the  brightest  and  darkest  regions  in  the  enhanced  image.  Thus, 

IMSHADE  has  achieved  a compromise  between  strictly  local  and  global 
normalization  techniques.  The  values  of  6 , $-,-,,01,,,  and  a control  the 
tradeoff  that  must  be  made  in  the  calculation  of  normalization  bounds. 

Extension  to  Two  Dimensional  Image  Functions 

The  same  steps  introduced  in  the  simplified  one  dimensional 
example  carry  over  into  the  two  dimensional  case.  The  image  is  sub- 
divided into  256  subimages.  Thus,  for  a 102b  x 800  image,  each  subimage 
is  6b  x 50  pixels.  Ceiling  and  floor  arrays  are  once  again  obtained  by 
extracting  the  maximum  and  minimum  intensity  from  each  subimage.  The  two 
l6  x l6  matrices  thereby  created  form  a two  dimensional  envelope  of  the 
image  function.  Each  array  is  expanded  at  its  boundaries  to  become  an 
18  x 18  matrix  by  copying  the  original  boundary  values  into  the  new  boundary 
elements.  This  step  is  required  for  the  nonlinear  spatial  smoothing  defined 
by  equations  3.1  to  3*3.  The  contrast  expansion  operation  which  was 
illustrated  in  Figure  3-15  is  defined  in  equations  3.^  and  3.5  for  the  two 
dimensional  problem. 

The  four  point  interpolation  required  to  obtain  piecewise  con- 
tinuous ceiling  and  floor  functions  are  defined  by  equations  3.6  to  3.8  and 
illustrated  by  Figure  3-18.  The  final  expression  for  the  normalized  image 
function  is  given  by  equation  3.9-  The  variables  W and  B select  the  gain 
and  bias  levels  appropriate  for  the  display. 


k 


Figure  3-18.  Interpolation  of  Ceiling  Array 


TWO  DIMENSIONAL  PROBLEM  EQUATIONS 


COARSE  ARRAYS  C(i,j)  AND  F(i,j)  SPATIAL  SMOOTHING  C'(i,j),  F'(ij) 


C ( i , j ) = | j C(i-l,j-l)+C(i,j-l)+C(i+l,j-l)+C(i-l,j ) 
+C(i+l,j)+C(i+l,J-l)+C(i+l,j )+C ( i+1 , j + 1 )| 
C(i,j )+C(i, j ) 


C' (i,j)  = MAX  { 
Similarly: 

F ' ( i , j ) = MIN  { 


1 + a 


, C(i,j)} 


a F(i,j)+F(i,j) 

-——a ’ F(i’^)} 


(3.1) 


(3.2) 


(3.3) 


CONTRAST  EXPANSION:  C"(i,j),  F"(i,j) 

C"(i,j)  = BcC'(i,j)  + (1-0C)  • MAX  (C) 

F"(i,j)  = BFF’(i,j)  + (l-ep)  -MAX  (F') 


where 


MAX ( C ' ) = MAX( C'(0,0),C'(l,0),... 
MAX(F')  = MAX(F' (0,0) ,F' (1,0) , . . . 


, C' (15,15)  ) 

, F' (15,15)  ) 


CONTINUOUS  CEILING  AND  FLOOR  FUNCTIONS  0 (k ,£ ) r- (k ,£ ) 


(3.h) 

(3.5) 


i = MOD ( k ) 

m = R(k) 

where  MOD  =>  Modulo  32 

j = MOD ( £ ) 

n = R(£) 

R =>  remainder 

Ax=  C(i,j)  + [_C(  i+1 , j ) - 

C(i,j  )]• 

(3.6) 

A 2=  C( i , j+1 )+ [c( i+1, j+1 ) 

- C (i  , j+1 )] * ^ 

(3.7) 

£(k,£)  = k±  + (A2-A1)  ■ 

~2  (See  Figure  3-18) 

(3.8) 

Similar ly,o^(k,£)  = A^  + (A^-A^)  • ~ if  F is  substituted  for  C in  equations 
3 . 6 and  3.7. 

I'(M>  • (W-B)  * B (3’9> 

where  W and  B are  the  overall  display  brightness  level  and  intensity  bias 
under  control  of  the  operator. 
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IMS HADE's  Variables  Ct  and  8 


The  IMSHADE  algorithm,  as  has  been  explained  in  the  simplified 


example  and  detailed  equations  of  the  last  section,  performs  filtering  on 
two  l6xl6  matrices  called  the  ceiling  and  floor  arrays.  The  parameters  ctf 
and  a control  the  degree  of  spatial  smoothing  and  g_  and  g control  the 
degree  of  contrast  enhancement.  Figures  3-20  through  3-2 2 illustrate  the 
effects  several  combinations  of  settings  have  on  the  original  cross- 
sectional  plot  from  Figure  3-19.  This  example  was  constructed  such  that 
each  subimage  from  the  l6xH00  full  image  is  1x25  pixels.  This  dimension 
is  shown  in  Figure  3-19- 


The  most  extreme  effects  achievable  with  IMSHADE  are  possible 

with  no  spatial  filtering  ( Ot  = a = 0)  and  full  contrast  expansion  (g  = g =l). 

or  L r 

This  case  is  shown  for  our  example  in  Figure  3-21.  Even  in  the  nearly  uni- 
formly bright  region  on  the  lefthand  side  of  the  original  image,  the  full 
contrast  enhancement  drives  the  local  minimum  down  to  the  global  minimum. 


This  nearly  total  loss  of  global  background  brightness  has 
perceptual  disadvantages  similar  to  those  of  AC  coupled  FLIR  systems. 

Most  of  the  examples  of  IMSHADE  found  in  Section  2 were  obtained 

with  no  spatial  smoothing  (a  =a . =0)  and  extensive,  but  not  full,  contrast 

C r 

expansion.  Figure  3-20  illustrates  these  conditions  for  our  example. 

Local  contrast  is  increased,  but  local  minima  and  maxima  do  not  reach  the 
global  extremes  reached  in  Figure  3-21. 
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Conclusions : 


The  FLIR  imagery  available  for  this  study  did  not  have  large  back- 
ground intensity  changes  across  the  image  and  therefore  did  not  exhaust  the 
display  dynamic  range.  In  fact,  increasing  local  feature-background  contrast 
tended  to  clutter  the  FLIR  scene  away  from  the  target.  This  effect  can  be 
observed  from  a comparison  of  the  pre-  and  post-enhancement  of  a scene  in 
Figures  2-3  and  2-5.  It  suggests  that  the  setting  of  the  contrast  expansion 
variables  8_  and  $ be  adaptively  determined  based  upon  the  dynamic  range  of 

U r 

the  background  intensity.  Thus,  the  tradeoff  setting  between  normalization 
to  local  or  to  global  extremes  performed  by  the  3 parameters  would  be  opti- 
mized adaptively  according  to  the  scene's  content.  One  approach  would  be 
based  upon  the  dynamic  ranges  of  the  256  local  maxima  in  the  ceiling  array 
and  the  256  local  minima  in  the  floor  array. 
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L(  CAL  CONTRACT  H ORMAL I ZAT TON 


Slices  through  a l6xA00  image  illustrate  effect  of  algorithm's  parameter 
values  . 
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3.2.2  Contouring 

The  FLIR  image  enhancement  potential  of  contouring  can  be 
justified  from  several  considerations.  First  of  all,  contouring  of  hot 
objects  against  cold  backgrounds  has  perceptual  advantages  similar  to 
edge  enhancement.  Secondly,  if  hot  targets  are  displayed  at  near  satur- 
ation levels,  selective  contouring  of  a high  temperature  band  can  high- 
light details  that  would  be  imperceptible  otherwise. 

The  image  degradation  potential  of  contouring  must  also  be 
considered  .however.  Targets  can  easily  be  hidden  among  too  many 
contour  lines.  Low  amplitude  noise  is  capable  of  generating  false  and 
partial  contour  lines,  as  are  many  simple  contouring  algorithms. 

Contour  Algorithm  (Parameters  = N,  d) 

A nearest  neighbor  algorithm  is  used  to  construct  contour  lines 
from  selected  image  points.  The  image  points  sought  are  those  at  the 
transition  of  the  image  intensity  function  through  pre-specified  contour 
levels.  The  contour  levels  are  uniformly  spaced  by  an  input  parameter  to 
be  referred  to  as  the  contour  interval  (d). 

Three  steps  are  required  to  decide  if  an  image  point  is  selected 
to  be  part  of  a contour  line. 

Step  1 - Nearest  neighbor  ranking.  The  eight  nearest  neighbors 
are  ranked  in  order  of  decreasing  intensity.  The  values  of  the  and 
( 8— N ) ^ ^ ranked  neighbor  are  specified  as  variable  "MAX"  and  "MIN". 
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Step  2 - Contour  level  transition  determination.  The  contour  levels 
above  which  MAX  and  MIN  reside  are  determined  by  integer  division  MAX/d  and 
MIN/d.  If  these  quotients  differ,  then  the  image  point  is  a candidate  for 
contour  line  formation. 

Step  3 - Noise  filtering.  Two  additional  tests  are  applied  to  image 
points  which  satisfy  Step  2's  contour  level  transition  test. 

1.  If  the  image  point  is  a minimum  or  a maximum  with  respect  to 
MAX  and  MIN  from  Step  1,  it  is  rejected  as  a contour  line  point. 

2.  If  the  difference  between  the  image  point  and  MAX  is  less  than 
5%  of  the  contour  interval,  the  image  point  is  not  classified 
as  a contour  line  point. 


If  a pixel  and  its  neighbors  satisfy  these  tests,  then  the  pixel's 
image  point  value  is  replaced  by  the  largest  permissible  value. 


[ ; 


Application 

Attempts  to  generate  contour  lines  on  FLIR  images  met  with  very 
limited  success  and  therefore  an  evaluation  of  its  enhancement  merits  is 
impossible.  Figures  2-65  and  2-68  illustrate  the  results  of  two  experiments. 
The  original  FLIR  images  data  underwent  local  two-dimensional  bandpass 
filtering  (Figures  2-6L  and  2-67)  prior  to  processing  by  the  contour  algorithm. 
This  prefiltering  sought  to  remove  noise  and  to  enhance  edges.  It  produced 
undesirable  blotchiness  in  the  background,  however,  and  caused  false  contour 
in  the  contoured  representations  of  the  two  scenes. 
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U . 0 HARDWARE  IMPLEMENTATION 

Second  generation  FLIR  systems  will  perform  some  signal  processing 
on  the  focal  plane  itself.  For  example,  the  background  suppression 
function  required  of  all  IR  imaging  systems  in  order  to  overcome  the 
enormous  D.C.  background  radiation  level  will  be  performed  on  the  focal 
plane.  On  focal  plane  multiplexors  will  generate  a TV  compatible  video 
signal.  Potential  mechanizations  of  these  functions  have  been  developed 
at  Rockwell  and  reported  in  a proposal  entitled  "Second  Generation  FLIR 
Image  Enhancement,"  (Report  No.  T76-257/501) . 

The  need  for  cryogenic  cooling  of  focal  plane  electronics,  however, 
restricts  the  complexity  (i.e.,  power  dissipation)  of  functions  integrated 
with  the  detector  array.  Therefore,  signal  processing  functions  which 
accept  and  generate  TV  compatible  video  signals  are  better  suited  to  off- 
focal  plane  mechanizations.  The  image  restoration  and  enhancement  techniques 
studied  under  contract  fall  into  this  class.  Although  no  detailed  analysis 
of  hardware  implementations  of  the  nearest  neighbor  and  dynamic  contrast 
normalization  algorithms  has  been  done,  they  possess  the  attributes  desired 
of  inline  video  processing  circuits: 

(1)  Video  input  and  video  output 

(2)  Real-time  performance 

(3)  Modest  storage  requirements 
(i.e.,  much  less  than  full  frame) 

1.1  Nearest  Neighbor  Implementation 

Most  of  the  specific  nearest  neighbor  algorithms  evaluated  were 
directed  at  the  particular  salt  and  pepper  noise  present  in  currently  available 
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FLIR  data,  and  thus  their  relevance  to  next  generation  FLIR  systems  is  open 
to  question.  As  a class,  however,  these  methods  can  perform  a wide  range 
of  linear  and  nonlinear  filtering  operations  including  gradient  based  edge 
enhancement  and  contouring. 

'v 

A generic  schematic  of  nearest  neighbor  algorithm  mechanizations  is 

shown  in  Figure  4-1.  Three  shi*ft  registers,  each  of  which  has  as  many 

stages  as  resolution  elements  in  the' TV  scan  line,  satisfy  the  storage 

requirements.  The  block  entitled  "Nearest  Neighbor  Algorithm  Circuit" 

performsthe  specific  function  desired,  such  as  replacement  of  cell  I 's 

■K.  9 Jb 

value  by  the  average  of  its  eight  neighbors  if  a specific  condition  is 
satisfied  (e.g.,  I . is  a local  extrema).  The  time  relationship  between 

K 9 jL 

the  incoming  and  outgoing  video  is  a fixed  delay  of  two  scan  line  periods . 
Since  1^  ^ in  Figure  4-1  is  simply  a filtered  version  of  1^  delayed  by 
two  scan  periods,  it  can  become  the  input  to  another  nearest  neighbor 
algorithm  circuit  or  to  another  video  processor  such  as  the  dynamic  contrast 
normalization  circuit  to  be  discussed  next. 

4.2  Dynamic  Contrast  Normalization 

The  dynamic  contrast  normalization  algorithm  is  suitable  for  off 
focal  plane  implementations  as  an  in  line  video  processor.  Its  computation 
and  storage  requirements  are  modest.  The  generic  block  diagram  for  the  class 
of  enhancement  algorithms  to  which  IMSHADE  belongs  is  found  in  Figure  4-2. 

The  block  entitled  "P  Parameter  Measurement  and  Storage"  monitors  the  video 
channel  and  calculates  P regional  and  global  dependent  variables  on  a frame 
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Figure  U-l  Nearest  Neighbor  Algorithms  Implementation 


by  frame  basis.  The  values  computed  for  the  n-1  th  frame  are  used  to 
filter  the  video  in  the  ntJl  frame.  In  the  figure,  k of  the  P parameters 
are  required  by  the  "Enhancement  Algorithm  Filter"  block  to  update  each 
pixel  value.  This  block  diagram  is  applicable  to  any  in  line  video 
processing  technique  which  can  tolerate  a one  frame  delay  between 
measurement  of  filter  variables  and  their  application  to  the  video  signal. 

A more  detailed  block  diagram  of  one  implementation  of  the 
dynamic  contrast  normalization  technique  is  found  in  Figure  **-3-  The  51** 
parameters  are  the  global  maximum,  the  global  minimum,  the  256  subimage 
maxima,  and  the  256  subimage  minima.  The  measurement  and  storage  circuitry 
are  shown  in  block  diagram  form  in  Figures  **-**  and  **-5. 

Two  dimensional  interpolation  circuitry  is  required  to  generate 
continuous  ceiling  and  floor  functions  for  the  final  normalization 
operation.  A simple  mechanization  is  described  in  Appendix  A. 

U.3  Interframe  Averaging  Techniques 

Inter-frame  temporal  integration  requires  that  a multiple  frame 
storage  register  be  utilized  to  store  consecutive  complete  IR  frames.  In 
order  to  realize  any  MRT  improvement  at  all  with  this  technique  requires 
that  more  than  six  IR  frames  be  stored  for  a 30  fps  system  (because  the  eye 
itself  temporally  integrates  for  0.2  seconds).  What  are  the  implications 
on  the  required  storage  capacity?  A second  generation  system  will  display 
as  many  as  10^  pixels  per  frame.  A factor  of  2 increase  in  MRT  at  the  eye 
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requires  that  2k  frames  of  a 30  fps  system  be  integrated  in  the  storage 
register.  A 100:1  dynamic  range  would  require  7 -tits  storage  capability 

Q 

per  pixel.  Thus,  a digital  memory  having  1.68  x 10  bit  storage  capacity 
would  be  required  for  a factor  of  2 increase  in  MRT. 

The  above  discussion  assumes  a stationary  image  is  being  viewed 
by  the  FLIR.  If  2k  frames  are  being  stored  and  summed  before  display, 
then  a moving  image  would  cause  considerable  smear  and  loss  of  resolution. 
This  loss  of  resolution  due  to  image  motion  would  have  to  be  traded  off 
against  the  increased  sensitivity. 
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5.0  CONCLUSIONS  AND  RECOMMENDATIONS 

The  locally  dependent  contrast  normalization  technique  is  an  excellent 
means  for  freeing  the  operator  from  continual  in-flight  display  adjustments. 

A full  evaluation  on  FLIR  imagery  was  not  possible  since  the  available  imagery 
data  did  not  contain  sufficient  dynamic  range  (s/N)  to  exhaust  the  dynamic 
range  of  the  display  and  fully  stress  the  enhancement  algorithm. 

A significant  portion  of  the  test  imagery  had  a target-background  to 
peak-to-peak  noise  ratio  of  less  than  1.  Noise  filtering  was  effectively 
accomplished  by  a combination  of  nonlinear  salt  and  pepper  noise  removal  and 
frequency  domain  bandstop  filtering.  The  application  of  these  techniques 
resulted  in  S/N  ratio  improvements  of  up  to  20  db. 

The  image  enhancement  and  noise  filtering  techniques  investigated  are 
practical  and  real  time  implementation. 


Significantly  improved  results  in  image  enhancement,  close  to  those 
depicted  in  non-FLIR  imagery,  Figures  2-48,  2-53  and  2-56,  can  be  expected 
with  advancement  in  focal  plane  technology  and  when  signals  obtained  closer 
to  the  focal  plane  are  available. 


A three  component  effort  of  further  image  enhancement  processing  is 
recommended  consisting  of:  preliminary  real-time  hardware  based  enhancement 

algorithm  optimization/evaluation,  expanded  enhancement  (cueing  preprocessing) 
studies  by  simulation  techniques,  and  systems  performance  prediction  evaluation 
study. 
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The  hardware  based  enhancement  algorithm  evaluation  would  serve  two 
functions:  first,  to  provide  a firm  link  between  laboratory  based  imare 

enhancement  performance  predictions  and  field  performance  figures  to  be  used 
in  the  full  blown  performance  model,  and  second,  act  as  a means  for  collecting 
data  for  further  image  processing  research  and  development  efforts. 

The  expanded  image  enhancement  task  should  investigate  additional 
processing  techniques  specifically  aimed  at  FLIR  type  data  as  well  as  applying 
to,  and  merging  with,  the  image  enhancement  and  target  cueing  processing. 
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